Qualitative, quantitative, and data mining methods for analyzing log data to characterize students' learning strategies and behaviors [discussant]

This symposium addresses how different classes of research methods, all based upon the use of log data from educational software, can facilitate the analysis of students’ learning strategies and behaviors. To this end, four multi-method programs of research are discussed, including the use of qualitative, quantitative-statistical, quantitative-modeling, and educational data mining methods. The symposium presents evidence regarding the applicability of each type of method to research questions of different grain sizes, and provides several examples of how these methods can be used in concert to facilitate our understanding of learning processes, learning strategies, and behaviors related to motivation, meta-cognition, and engagement

New measurement paradigms

This collection of New Measurement Paradigms papers represents a snapshot of the variety of measurement methods in use at the time of writing across several projects funded by the National Science Foundation (US) through its REESE and DR K–12 programs. All of the projects are developing and testing intelligent learning environments that seek to carefully measure and promote student learning, and the purpose of this collection of papers is to describe and illustrate the use of several measurement methods employed to achieve this. The papers are deliberately short because they are designed to introduce the methods in use and not to be a textbook chapter on each method. The New Measurement Paradigms collection is designed to serve as a reference point for researchers who are working in projects that are creating e-learning environments in which there is a need to make judgments about students’ levels of knowledge and skills, or for those interested in this but who have not yet delved into these methods

Towards Scalable Assessment of Performance-Based Skills: Generalizing a Detector of Systematic Science Inquiry to a Simulation with a Complex Structure

Abstract. There are well-acknowledged challenges to scaling computerized performance-based assessments. One such challenge is reliably and validly identifying ill-defined skills. We describe an approach that leverages a data mining framework to build and validate a detector that evaluates an ill-defined inquiry process skill, designing controlled experiments. The detector was originally built and validated for use with physical science simulations that have a simpler, linear causal structure. In this paper, we show that the detector can be used to identify demonstration of skill within a life science simulation on Ecosystems that has a complex underlying causal structure. The detector is evaluated in three ways: 1) identifying skill demonstration for a new student cohort, 2) handling the variability in how students conduct experiments, and 3) using it to determine when students are off-track before they finish collecting data

Looking inside the black box: assessing model-based learning and inquiry in BioLogica TM

Abstract: The Modeling Across the Curriculum Project (MAC; IERI # 0115699, Oct 2001-2006) used real-time assessments to facilitate student learning and model-based inquiry among high school students. We developed technology, materials, and processes that enabled us to monitor and respond to students' actions. MAC learning activities engage students in a progressive model-building approach Looking inside the black box 16

Discovery with Models: A Case Study on Carelessness in Computer-based Science Inquiry

In recent years, an increasing number of analyses in Learning Analytics and Educational Data Mining (EDM) have adopted a "Discovery with Models" approach, where an existing model is used as a key component in a new EDM/analytics analysis. This article presents a theoretical discussion on the emergence of discovery with models, its potential to enhance research on learning and learners, and key lessons learned in how discovery with models can be conducted validly and effectively. We illustrate these issues through discussion of a case study where discovery with models was used to investigate a form of disengaged behavior, i.e., carelessness, in the context of middle school computer-based science inquiry. This behavior has been acknowledged as a problem in education as early as the 1920s. With the increasing use of high-stakes testing, the cost of student carelessness can be higher. For instance, within computer-based learning environments careless errors can result in reduced educational effectiveness, with students continuing to receive material they have already mastered. Despite the importance of this problem, it has received minimal research attention, in part due to difficulties in operationalizing carelessness as a construct. Building from theory on carelessness and a Bayesian framework for knowledge modeling, we use machine-learned detectors to predict carelessness within authentic use of a computer-based learning environment. We then use a discovery with models approach to link these validated carelessness measures to survey data, to study the correlations between the prevalence of carelessness and student goal orientation. The second construct, carelessness, refers to incorrect answers given by a student on material that the student should be able to answer correctly Rodriguez-Fornells & Maydeu-Olivares, 2000). The application of discovery with models involves two main phases. First, a model of a construct is developed using machine learning or knowledge engineering techniques, and is then validated, as discussed below. Second, this validated model is applied to data and used as a component in another analysis: For example, for identifying outliers through model predictions; examining which variables best predict the modeled construct; finding relationships between the construct and other variables using correlations, predictions, associations rules, causal relationships or other methods; or studying the contexts where the construct occurs, including its prevalence across domains, systems, or populations. For example, in One essential question to pose prior to a discovery with model analysis is whether the model adopted is valid, both overall, and for the specific situation in which it is being used. Ideally, a model should be validated using an approach such as cross-validation, where the model is repeatedly trained on one portion of the data and tested on a different portion, with model predictions compared to appropriate external measures, for example assessments made by humans with acceptably high inter-rater reliability, such as field observations of student behavior for gaming the system (cf. Even after validating in this fashion, validity should be re-considered if the model is used for a substantially different population or context than was used when developing the model.. An alternative approach is to use a simpler knowledge-engineered definition, rationally deriving a function/rule that is then applied to the data. In this case, the model can be inferred to have face validity. However, knowledge-engineered models often DISCOVERY WITH MODELS: A CASE STUDY ON CARELESSNESS 6 produce different results than machine learning-based models, for example in the case of gaming the system. Research studying whether student or content is a better predictor of gaming the system identified different results, depending on which model was applied (cf. Baker, 2007a

Relative Roles of the Cellular and Humoral Responses in the Drosophila Host Defense against Three Gram-Positive Bacterial Infections

BACKGROUND: Two NF-kappaB signaling pathways, Toll and immune deficiency (imd), are required for survival to bacterial infections in Drosophila. In response to septic injury, these pathways mediate rapid transcriptional activation of distinct sets of effector molecules, including antimicrobial peptides, which are important components of a humoral defense response. However, it is less clear to what extent macrophage-like hemocytes contribute to host defense. METHODOLOGY/PRINCIPAL FINDINGS: In order to dissect the relative importance of humoral and cellular defenses after septic injury with three different gram-positive bacteria (Micrococcus luteus, Enterococcus faecalis, Staphylococcus aureus), we used latex bead pre-injection to ablate macrophage function in flies wildtype or mutant for various Toll and imd pathway components. We found that in all three infection models a compromised phagocytic system impaired fly survival--independently of concomitant Toll or imd pathway activation. Our data failed to confirm a role of the PGRP-SA and GNBP1 Pattern Recognition Receptors for phagocytosis of S. aureus. The Drosophila scavenger receptor Eater mediates the phagocytosis by hemocytes or S2 cells of E. faecalis and S. aureus, but not of M. luteus. In the case of M. luteus and E. faecalis, but not S. aureus, decreased survival due to defective phagocytosis could be compensated for by genetically enhancing the humoral immune response. CONCLUSIONS/SIGNIFICANCE: Our results underscore the fundamental importance of both cellular and humoral mechanisms in Drosophila immunity and shed light on the balance between these two arms of host defense depending on the invading pathogen

diagramming versus explaining.

The effects of different learning tasks on model-building in Plate Tectonics